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This paper reports on a study that examines the effects of problem context on students’ 
performance. The performance of 151 Year 10 students on six mathematical problems was 
compared with the performance on fifteen variants with more and less context familiarity (CF) 
and engagement (CE) across levels of context use (LCU). The latter explanatory variables (CF, 
CE, and LCU) are used to estimate the strength of the relationship among them and the students’ 
performance. Results show that neither CF nor CE affect students’ performance but LCU 
demanded in solving a problem does. 


The incorporation of context in mathematical problems has been highly recommended by 
current refonn documents and mathematics curricula around the globe (e.g., OECD, 2013) and 
this has led to the development of new fonns of connectedness of the instructional 
mathematical content by focussing on solving problems in context and mathematical 
modelling (Stillman, 2000). Nevertheless, in this escalating emphasis on problems set in 
context, the role that problem context plays on students’ performance seems to be an unsolved 
matter. The fact that the influence of context on students’ performance in mathematical 
problems is a matter that cannot be disregarded is confirmed by ample research. For instance, 
this author agrees with De Lange (2007) that: 

The influence of contexts should be studied much more systematically than is presently the case, and we 
researchers should refrain from strong statements that we have proven to be of disputable quality until 
we have firmer evidence (De Lange, 2007, p. 1 120). 

In general, evidence is sparse. The knowledge of the findings of individual studies 
highlights that there is a lack of a firm body of convincing empirical evidence for the effects 
(in any direction) of the context of a problem on students’ performance (Stacey, 2015). Thus, 
for this paper, the purpose is therefore: to report an empirical study on the possible effect of an 
alteration of problem context familiarity and engagement on the perfonnance of Year 10 
students. 


Key Terminology 


(Problem ) Context 

In this study context refers to the information that surrounds a scenario described in a 
problem that needs to be mathematised. The surrounding infonnation might be necessary or 
unnecessary for the mathematisation of the problem, but this is independent from the 
problem’s syntax and the stimulus. 

To clarify the intended definition of context and the distinctions above, consider Problem 1 
provided in Table 1. 
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Table 1 

Problem examples 

Problem 0 : Some buyers are looking at the technical infonnation sheets of similar spin 
dryer machines A and B: Dryer machine A’s drum spins 17 times every 10 seconds, Dryer 
machine B’s drum spins 35 times every 21 seconds. Which dryer spins faster? 

Problem 1 : John works full-time, 5 days a week. He spends $45 weekly on lunch with a 
drink included. He finds a very similar place to have lunch with no drinks included; If he 
decides to have lunch there, he will spend only $35 weekly on lunch but he will have to 
spend an extra $50 monthly on drinks. Based on this infonnation, should John have his 
lunch at his regular place or try the new one? 

Problem 2 : For a rock concert, a rectangular field of size 100 m by 50 m was reserved for 
the audience. The concert was completely sold out and the field was full with all the fans 
standing. Which one of the following is likely to be the best estimate of the total number of 
people attending the concert?* 

A) 2 000 B) 5 000 C) 20 000 D) 50 000 E) 100 000 

• Source OECD (2006, p. 94) 

The context for Problem 1 relates to having lunch when working on a full-time job. For 
example, context involves general aspects such as value for money and the number of working 
weeks of a full-time job in a calendar year to solve the problems. Here, for the 
mathematisation of the problem this context is needed for students to realise that a full-time 
job week in the state of Victoria, Australia, has 5 days and 48 weeks respectively. In this 
problem, the words and the gramatical structure give the syntax, whereas the stimulus is the 
actual physical layout written on the page, without diagrams. 

Levels of Context Use ( LCU ) 

LCU was introduced by De Lange in 1979. However, the definitions provided by De Lange 
(1979) need clarification; De Lange’s first and second order uses of context blur because in the 
examination of their descriptions essential differences between them are difficult to be 
distinguished. In a previous work from the author of this paper (not yet published and not 
reported here), LCU was clarified, subjected to statistical validation, and re-defined as follows: 

• Zero order use of context : At this level, context provides the opportunity to take direct 
actions or make direct inferences from the instructions given in a mathematics problem. 
Therefore, context of a problem is not used to interpret mathematical results or arguments. 

• First order use of context. At this level, context is used to either identify or select relevant 
infonnation, variables, or relationships for the mathematical formulation of a problem. Also, 
context is used to detennine the adequateness of the mathematical results. 

• Second order use of context At this level, context is used to either define or retrieve relevant 
variables, relationships, or assumptions for the mathematical formulation of a problem. Also, 
context is used to judge the adequacy of the mathematical results and arguments in terms of 
the original problem. 

Under this new classification of LCU, problems presented in Table 1 were classified as 
follows: Problem 0 in Table 1 was classified as zero order. Here, the context provides a chance 
to take direct actions for fonnulating the problem in mathematical terms. It is expected that 
Year 10 students know how to use the particular mathematical model needed (i.e., compare 
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ratios) in a range of contexts. Neither assumptions nor a new mathematical model nor 
interpretation of the results are required, hence the emphasis is on the mathematics involved 
but not on the problem context. Problem 1 was classified as first order. The context of the 
problem is needed to mathematise the problem. Students have to identify relevant information 
(e.g., number of weeks of a full-time job) to fonnulate mathematically the problem. It is 
considered that results do not need to be further interpreted or evaluated against the context. 
Lastly, Problem 2 was classified as second order. The context of this problem is required to 
find the number of people that can be accommodated per square metre. Therefore, context 
provides a chance to define assumptions and constraints to use a model and validate the 
answer in relation to the context. 

For this research, LCU is used as an analytical tool to analyse how the context of a 
problem is used to fonnulate a problem in mathematical terms and to interpret the answer in 
relation to the context of a given problem. 

Context Familiarity ( CF) and Context Engagement ( CE) 

A search for definitions of context familiarity revealed unexpectedly that the existing 
literature does not provide an unambiguous understanding of the term context familiarity, 
although its effects on the perfonnance of students are researched and reported. Apparently, 
context familiarity is often implicitly meant by researchers as students’ close knowledge or 
experience. Hence, in this study, a familiar context is drawn from students’ everyday 
experiences. Correspondingly, non- familiar contexts in this study mean that problem contexts 
are not drawn from the students’ everyday experiences. 

Context engagement of a problem is understood as a students’ desire to solve a 
mathematical problem within a certain context. Thus, a more engaging context for students in 
this study means that the context results in a desire for students to solve the problem. 
Correspondingly, non- engaging context in this study refer to contexts that do not result in a 
desire for to students to solve the problem. 

The Study 

In the study, the effects of context familiarity and engagement across LCU on students’ 
performance are investigated. Subjects were from eight Year 10 classes (n = 151, 76 males and 
75 females) at one volunteer co-educational selective entry secondary school in Melbourne, 
Australia. Data collection occurred in Term 4, 2014. Two experiments were devised to observe 
whether the nature of determination of CF and CE may increase, decrease or maintain the 
same level of students’ performance across problems between experiments. In the first 
experiment context familiarity and engagement is determined from the researcher’s opinion, 
whereas in the second experiment, these factors (i.e., CF and CE) were determined from the 
students’ preferences to certain contexts using a student background survey (SBS). In both 
experiments, the problem context was varied whilst holding constant other features of the 
problem that are known to affect performance (perfonnance is interpreted as average percent 
correct, score out of 2 points). There are two research questions: 

RQ1: Is students’ performance better on problems with more familiar and engaging 
contexts? 

RQ2: What is the relative effect of context familiarity, engagement and levels of context 
use on students’ perfonnance? 
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Characteristics and Creation of Problems Used in the Two Experiments 

Six problems (two at each LCU) were prepared and used as pivots to create fifteen extra 
variants on context. Pivot problems’ contexts were not varied so they exhibit less context 
familiarity and engagement for students. Variant problems are characterised by the same 
mathematical core (i.e., same mathematical content, competency and model) and LCU of their 
corresponding pivot, but they differ in the context in which the problem is embedded. The 
context in which variants are embedded exhibits more context familiarity and engagement for 
students (either assumed or empirically detennined) in both experiments. 

To exemplify the latter, Table 2 shows one set of variants on context, from the first 
experiment, constructed from problems presented in Table 1. 

Table 2 

Example of one set of variants constructed from problems in Table 1 

Problem AO : Some children are making pink paint by mixing together white and red. Maria 
uses 17 spoonfuls of red paint and 10 spoonfuls of white paint, Marc uses 35 spoonfuls of 
red paint and 2 1 spoonfuls of white paint. Whose paint will be darker pink? 

Problem Al : John attends primary school 5 days a week. His mother spends $45 weekly on 
a lunch-box with a drink included for John. John’s mother finds a very similar lunch-box at 
a local supermarket but with no drinks included; if she decides to buy lunch for John there, 
she will have to spend only $35 weekly on the lunch-box but she will have to spend an 
extra $50 monthly on drinks. Based on this infonnation, should John’s mother buy the 
lunch-box at the regular place or try the new one? 

Problem A2 : In a ferry, a rectangular space of size 50 m by 25 m was reserved for 
transporting cars. The ferry was full with cars. Which one of the following is likely to be 
the best estimate of the total number of cars being transported in the ferry? Show your 
work and explain your reasoning. 

A) About 60 B) About 125 C) About 180 D) About 200 


Table 3 presents the distribution in frequency of the twenty-one problems used for data 
collection per experiment. As can be seen from Table 3, the number of variants in the second 
experiment is not consistent. 


Table 3 

Type, frequency and distribution per experiment of problems used for data collection 


Level of Context Use 


First experiment 

Second experiment 

Total 

Pivot 

Variant 

Pivot 

Variant 

Zero 

2 

2 

2 

2 

8 

First 

2 

2 

2 

0 

6 

Second 

2 

2 

2 

1 

7 

Total 

6 

6 

6 

3 

21 


Two reasons explain this disparity. Firstly, three contexts used in the first experiment 
matched the students’ preferences to contexts more or less familiar and engaging to them; 
hence, this researcher decided to use the same problems in context in the second experiment. 
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Secondly, creating variants on context was difficult, and reveals important aspects about how 
context is needed for solving the problem, especially for the second order use of context. 
Although the basic underlying mathematical core and LCU may be the same, a change in 
context introduces new assumptions and infonnation and this, sometimes, changed the 
problems, hence it was not possible to find a balance between pivot and variant problems at 
each LCU. In creating the matched pair, context was changed, but keeping the same 
mathematical core of the problem and, to the best extent possible, the problems’ syntax and 
stimulus were held constant (these are not discussed here). 

Participants and the Rotated Test Design 

The performance of one hundred and fifty one students (n=151) from eight Year 10 classes 
at one volunteer school in Melbourne, Australia, on fifteen problems was compared using 
paper-and-pencil tests in two experiments. Sixty three students (m=63) were tested in the first 
experiment, whereas eighty eight students (n 2 = 88 ) in the second experiment. All participants 
were tested in a class period. Because of the limited time for testing in a class period and 
potential inter-problem effects, a rotated design of eight booklets was used, so that students did 
not solve both a pivot and its variant. This means that direct comparison of performance at the 
level of the student was not possible. Each booklet contained three problems, either pivot or 
variant problems, at each LCU. Booklets were distributed at random to each student. 

The Preliminary Rating Study (SBS) 

Ratings on CF and CE, obtained from students, were used to guide the construction of 
problems for the second experiment. The same students who answered the SBS participated in 
the second experiment. Thirty contexts were presented in a questionnaire to students (n = 88 ) 
in the classes assigned to the second experiment. The questionnaire consisted of two sections 
(each containing the same thirty contexts), which explored separately students’ preferences to 
context familiarity and engagement. Contexts that received the lower and higher frequencies 
on the two sections of the questionnaire, containing a 5 -point Likert scale (scale covering from 
Not familiar at all to Extremely familiar, and Not engaging at all to Extremely engaging, 
correspondingly), were selected as less and more familiar and engaging contexts for students 
respectively. To observe whether the students’ ratings of the thirty contexts in the 
questionnaire all reliably measure the same latent variable (i.e., CF and CE, and therefore a 5- 
point Likert scale could be constructed); a Cronbach's alpha was run on the sample size 
(n= 88 ). A Cronbach’s alpha coefficient of the ratings’ scale on CF and CE was found to be 
0.793 and 0.761 respectively. Therefore, the scales can be considered reliable with the sample 
(Pallant, 2001) on determining CF and CE. 


Results 

Table 4 below presents summaries of the distribution of the students’ performance, in 
frequencies, as well as means and standard deviations of all the explanatory variables. The 
students’ performance was scored as No Credit, Partial Credit and Full Credit. Also, it offers a 
summary of the odds ratios of students obtaining Full Credit versus No Credit performance at 
one level of the ordinal variable. In general, overall performance across contexts more familiar 
and engaging for students (either assumed or determined) does not show large changes from 
problems set in contexts less familiar and engaging for students (either assumed or 
determined). Also, it can be seen that students’ performance decreased as LCU increased. 
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Table 4 

Frequency distribution of students’ performance by LCU, CF and CE, and a summary of the 
odds of students obtaining Full Credit versus No Credit performance at one level of the 


ordinal variable 



Student Performance 




Odds 

P 



No 

Partial 

Full 

Total 

Mean 

S.d 

Ratio 

value 



Credit 

Credit 

Credit 




(ORs) 


Second 

65 

52 

34 

151 

0.795 

0.786 

6.639 

0.000 

LCU 

First 

28 

63 

60 

151 

1.212 

0.736 

2.627 

0.000 


Zero 

(Reference) 

30 

14 

107 

151 

1.510 

0.807 

1 


Total 


123 

129 

201 

453 






Extremely 

Familiar 

21 

24 

24 

69 

1.043 

0.812 

1.358 

0.366 


Very 

Familiar 

21 

25 

28 

74 

1.095 

0.814 

1.332 

0.360 

Levels 

Moderately 

Familiar 

35 

25 

64 

124 

1.234 

0.866 

1.05 

0.860 

of CF 

Slightly 

Familiar 

19 

24 

42 

85 

1.271 

0.808 

0.994 

0.983 


Not at all 

Familiar 

(Reference) 

27 

31 

43 

101 

1.158 

0.821 

1 


Total 


123 

129 

201 

453 






Extremely 

Engaging 

10 

9 

12 

31 

1.065 

0.854 

0.652 

0.335 


Very 

Engaging 

7 

14 

11 

32 

1.125 

0.751 

0.833 

0.642 

Levels 

Moderately 

40 

37 

54 

131 

1.107 

0.844 

0.888 

0.648 

of CE 

Engaging 










Slightly 

Engaging 

27 

39 

75 

141 

1.340 

0.782 

0.536 

0.014 


Not at all 
Engaging 
(Reference) 

39 

30 

49 

118 

1.085 

0.863 

1 


Total 


123 

129 

201 

453 






To test Research Question 1, a Kruskal-Wallis test showed that there is not a statistically 
significant difference in the mean in students’ performance between pivot and variant 
problems (first experiment: x 2 (l) = 0.010, p=0.922, with a mean rank of pivot scores of 1 17.14 
for pivots and 1 17.94 for variants; second experiment: yf( 1 )=0.353, p=0.553, with a mean ra nk 
of 107.53 for pivots and 1 12.31 for variants). Therefore, the null hypothesis that performance 
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was equivalent on pivot and variant problems, in both experiments, is accepted. Further 
Kruskal- Wallis tests showed that there was not a statistically significant difference in students’ 
performance between pivot and variant problems (x 2 (l)=0.094, p=0.759), and levels of context 
familiarity (x 2 (4)=4.750, p=0.314) and engagement (x 2 (4)=8.420, p=0.077) from the first and 
second experiment. Considering these results and as pivot and variant problems have, to the 
best extent possible, the same mathematical core, syntax and stimulus, and were categorised 
into LCU using the same construct, but varied on the context in which they are set, so it was 
decided that pivot and variant problems from the two experiments could be blended for the 
next analyses. 

For Research Question 2 the relative effect of all the explanatory variables (i.e., CF, CE 
and LCU) on the students' performance is investigated. This quantification is needed, mainly, 
to have a clear picture of the impact of the research design on the students’ perfonnance. An 
ordinal logistic regression analysis is used to test Research Question 2. Data was collected 
from students using a 5-point-Likert scale for context familiarity (scale ranging from Not 
familiar at all=l to Extremely familiar=5) and a 5-point-Likert scale for context engagement 
(scale ranging from Not engaging at all=l to Extremely engaging=5). The other explanatory 
variable included levels of context use (coded as: zero level=0, first level=l, and second level= 
2). The dependent variable, students’ performance, was coded as No Credit=0, Partial 
Credit=l, and Full Credit=2. As every variable in the model to be reported below is a 
categorical variable (later coded as ordinal), the odds ratios refer to the odds of students 
obtaining Full Credit versus No Credit performance at one level of the ordinal variable relative 
to a baseline level of that variable (i.e., to the lowest ordinal code at each variable). 

Table 4 presents concisely the distribution of students’ scores across all the explanatory 
variables, as well as a summary of the ordinal regression containing the odds ratios (ORs), and 
the statistically significant p-values. In this sample, a test of the full model against a constant 
only model indicates that there is a statistically significant relationship between the predictors 
and the dependent variable (x 2 (10)=76.621, p=0.000). The Nagelkerke’s R 2 , an indicator of the 
goodness-of-fit of the model was 17.6%. Variables whose corresponding odds ratios are 
significantly greater than 1.0 have significant effects on the dependent variable (i.e., students’ 
performance) in the model. Among the explanatory variables, presented in Table 4, the 
following were significantly more likely to affect students’ performance in this sample: second 
order use of context (OR, 6.639, 95% Cl [2.685, 16.411]) and first order use of context (OR, 
2.627, 95% Cl [1.673, 4.125]). The model shows that the odds ratio of a student obtaining Full 
Credit versus No Credit perfonnance was 6.639 times higher on zero level of context use 
problems than on second level of context use problems, when the effects of other predictors 
(i.e., levels of context familiarity and engagement) are held constant (i.e., controlled). 
Similarly, the model shows that the odds ratio of a student obtaining Full Credit versus No 
Credit performance was 2.627 times higher on zero level of context use problems than on first 
level of context use problems, after controlling for the effects of other predictors. 

Discussion and Conclusion 

The present study was designed to investigate the effects of familiarity and engagement of 
problem context across levels of context use on the performance of Year 10 students. In 
particular, the context in mathematical problems was varied in CF and CE across LCU whilst 
holding constant other features of the problem that are known to affect perfonnance. To 
elucidate how the latter mentioned factors influence students’ perfonnance, two experiments 
were designed. The cohort of participants was divided into two groups according to how CF 
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and CE were determined. The first research question in this study sought to detennine the 
effect of familiar and engaging contexts on students’ performance. This study found that 
students did not perform better on contexts that were more familiar and engaging for them, 
suggesting that embedding mathematical problems in contexts that were more familiar and 
engaging for students does not necessarily enhance perfonnance. This is a surprising result 
given that mathematical problems in context are promoted especially for their supposed 
contribution to the use of students’ knowledge (interpreted here as context familiarity), 
students’ motivation, confidence, engagement and interest in and within a problem. The 
second research question was related to the relative effect of context familiarity, context 
engagement and levels of context use on students’ performance. Results showed that neither 
higher levels of CF nor higher levels of CE affected students’ performance but rather higher 
LCU. In the current study, a student obtaining Full Credit versus No Credit performance on a 
zero order use of context problem was approximately seven times higher than a second order 
use of context problem and three times higher than a first order use of context problem. It is 
not clear from the literature whether problem contexts, classified in terms of LCU, have such 
effect on students’ performance. Hence, this result must be interpreted with caution because of 
the type and number of second order use of problems used in this research. Only modelling 
and reasoning and argumentation were considered as the mathematical competencies used in 
the seven second order use of context problems. Creating variants on context was extremely 
difficult, especially for the second order use of context. Therefore, it was not possible to find a 
balance between pivot and variant problems at each level of context use. In addition, the latter 
influenced the smaller number of contexts and mathematical competencies included for this 
research. Because of this, it was not possible to have as a big picture as desired of the impact 
of context on students’ performance. As a result, with the small sample of problems and 
competencies used in this research, caution must be applied with the previous results, as these 
might not be transferrable to other contexts involving different mathematical competencies. 
Lastly, in order to add robustness to the methodology and result of the present study, a further 
large-scale quantitative study is needed to investigate the research questions posed for this 
study, but taking into account more mathematical competencies, problem fonnats, numbers of 
problems, and secondary students at different levels of instruction. 

References 

De Lange, J. (2007). Large-scale assessment and mathematics education. In F. Lester (Ed.), Second handbook of 
research on mathematics teaching and learning (pp.l 1 12-1 142) Charlotte, N.C.: Information Age 
Publishing. 

De Lange, J. (1979). Contextuele problemen [Contextual problems]. Euclides, 55, 50-60. 

Organisation for Economic Co-Operation and Development. (2013). PISA 2015 Draft mathematics framework. 
Retrieved January 6, 2015, from 

http://www.oecd.org/pisa/pisaproducts/Draft%20PISA%202015%20Mathematics%20Framework%20.pdf 
Organisation for Economic Co-Operation and Development. (2006). PISA released items -Mathematics. Retrieved 
May 22, 2013, from http://www.oecd.org/pisa/38709418.pdf 
Pallant, J. (2001). SPSS survival manual. Crows Nest NSW: Allen & Unwin. 

Stacey, K. (2015). The Real World and the Mathematical World. In K. Stacey & R. Turner (Eds.), Assessing 
Mathematical Literacy. The PISA Experience (pp. 57-84). Cham: Springer International Publishing. 

Stillman, G. (2000). Impact of prior knowledge of task context on approaches to applications tasks. The Journal 
of Mathematical Behavior , 19(3), 333-361. 


109 



